Leveraging AWGR-based Optical Packet Switches to Reduce Latency in Petascale Computing Systems

High Performance Computing Systems will have a mix of optical and electrical interconnects as demonstrated by the IBM PERCS (Programmable Easy to Use Reliable Computing Systems) in order to balance energy efficiency, scalability, and performance. In this paper we start with the PERCS design and show how network latency can be reduced by using optical packet switches in the system level interconnect. We compare AWGR-based optical switches which use a distributed control plane, AWGR-based optical switches which use centralized control planes and physical layer contention resolution, and the baseline PERCS-like design and show up to a 27% reduction in flit latency on some traffic patterns, even under high loads. We then leverage the reduction in hop-count offered by the top-level AWGR switches to (a) improve the lower layer interconnect by simplifying the HUB chip and/or reducing both the number of links and the bandwidth of the intra supernode interconnect, and (b) improve the scalability of the system by allowing more supernodes to be connected together.